Algebraic-Geometric and Probabilistic Approaches for Clustering and Dimension Reduction of Mixtures of Principle Component Subspaces ECE842 Course Project Report

Generalized Principal Component Analysis (GPCA) and Probabilistic Principal Component Analysis (PPCA) are two extensions of PCA approaches to the mixtures of principal subspaces. GPCA is an algebraic geometric framework in which the collection of linear subspaces is represented by a set of homogeneous polynomials whose degree corresponds to the number of subspaces and whose factors (roots) encode the subspace parameter. PPCA is a probabilistic approach where the principal component analysis is viewed as a maximum-likelihood procedure based on a probability density model of the observed data. Both techniques are capable of estimating a mixture of subspaces from sample data points, thus useful for data clustering and dimension reduction problems in multivariate data mining. The primary goal of this project is to carry out a conceptual study, to explore the principles and features of the algebraic-geometrical and probabilistic approaches to mixtures of principal component subspaces, and learn from hand-on experience though computational implementation of these techniques. A polynomial factorization algorithm (PFA) for GPCA and an expectation-maximization (EM) for PPCA were implemented using MATLAB codes. The implemented algorithms have been tested on synthetic data sets. It was shown that the PFA algorithm for GPCA can successfully identify the number of subspaces in the mixture, and estimate the normal vectors of the subspaces, if successful, with a relative high correlation. However, the implemented algorithm is not robust as it is data dependent. The potential problems of this implementation were discussed in the report. The implemented EM algorithm for PPCA showed that a probabilistic mixture model can identify the clusters, and assign the cluster association of each data point correctly. Both techniques estimated the component subspaces of lower dimensionality, thus data dimensions can be reduced and underlying clusters can be recovered. In this project, the implemented algorithms were only tested on synthetic 3-dimensional data and not yet tested on higher dimensional data or real data, and the algorithms are far from comprehensive for practical use. However, the computational implementation helped a lot in the understanding of the two approaches for mixtures of principal component subspaces.